{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5-6 评估指标metrics\n",
    "\n",
    "损失函数除了作为模型训练时候的优化目标，也能够作为模型好坏的一种评价指标。但通常人们还会从其它角度评估模型的好坏。\n",
    "\n",
    "这就是评估指标。通常损失函数都可以作为评估指标，如MAE, MSE, CategoricalCrossentropy等也是常用的评估指标。\n",
    "\n",
    "但评估指标不一定可以作为损失函数，例如AUC, Accuracy, Precision。因为评估指标不要求连续可导，而损失函数通常要求连续可导。\n",
    "\n",
    "编译模型时，可以通过列表形式指定多个评估指标。\n",
    "\n",
    "如果有需要，也可以自定义评估指标。\n",
    "\n",
    "自定义评估指标需要接收两个张量y_true, y_pred作为输入参数，并输出一个标量作为评估值。\n",
    "\n",
    "也可以对tf.keras.metrics.Metric进行子类化，重写初始化方法, update_state方法, result方法实现评估指标的计算逻辑，从而得到评估指标的类的实现形式。\n",
    "\n",
    "由于训练的过程通常是分批次训练的，而评估指标要跑完一个epoch才能够得到整体的指标结果。因此，类形式的评估指标更为常见。即需要编写初始化方法以创建与计算指标结果相关的一些中间变量，编写update_state方法在每个batch后更新相关中间变量的状态，编写result方法输出最终指标结果。\n",
    "\n",
    "如果编写函数形式的评估指标，则只能取epoch中各个batch计算的评估指标结果的平均值作为整个epoch上的评估指标结果，这个结果通常会偏离拿整个epoch数据一次计算的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 一、常用的内置评估指标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* MeanSquaredError（平方差误差，用于回归，可以简写为MSE，函数形式为mse）\n",
    "\n",
    "* MeanAbsoluteError (绝对值误差，用于回归，可以简写为MAE，函数形式为mae)\n",
    "\n",
    "* MeanAbsolutePercentageError (平均百分比误差，用于回归，可以简写为MAPE，函数形式为mape)\n",
    "\n",
    "* RootMeanSquaredError (均方根误差，用于回归)\n",
    "\n",
    "* Accuracy (准确率，用于分类，可以用字符串\"Accuracy\"表示，Accuracy=(TP+TN)/(TP+TN+FP+FN)，要求y_true和y_pred都为类别序号编码)\n",
    "\n",
    "* Precision (精确率，用于二分类，Precision = TP/(TP+FP))\n",
    "\n",
    "* Recall (召回率，用于二分类，Recall = TP/(TP+FN))\n",
    "\n",
    "* TruePositives (真正例，用于二分类)\n",
    "\n",
    "* TrueNegatives (真负例，用于二分类)\n",
    "\n",
    "* FalsePositives (假正例，用于二分类)\n",
    "\n",
    "* FalseNegatives (假负例，用于二分类)\n",
    "\n",
    "* AUC(ROC曲线(TPR vs FPR)下的面积，用于二分类，直观解释为随机抽取一个正样本和一个负样本，正样本的预测值大于负样本的概率)\n",
    "\n",
    "* CategoricalAccuracy（分类准确率，与Accuracy含义相同，要求y_true(label)为onehot编码形式）\n",
    "\n",
    "* SparseCategoricalAccuracy (稀疏分类准确率，与Accuracy含义相同，要求y_true(label)为序号编码形式)\n",
    "\n",
    "* MeanIoU (Intersection-Over-Union，常用于图像分割)\n",
    "\n",
    "* TopKCategoricalAccuracy (多分类TopK准确率，要求y_true(label)为onehot编码形式)\n",
    "\n",
    "* SparseTopKCategoricalAccuracy (稀疏多分类TopK准确率，要求y_true(label)为序号编码形式)\n",
    "\n",
    "* Mean (平均值)\n",
    "\n",
    "* Sum (求和)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 二、 自定义评估指标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以金融风控领域常用的KS指标为例，示范自定义评估指标。\n",
    "\n",
    "KS指标适合二分类问题，其计算方式为 KS=max(TPR-FPR).\n",
    "\n",
    "其中TPR=TP/(TP+FN) , FPR = FP/(FP+TN) \n",
    "\n",
    "TPR曲线实际上就是正样本的累积分布曲线(CDF)，FPR曲线实际上就是负样本的累积分布曲线(CDF)。\n",
    "\n",
    "KS指标就是正样本和负样本累积分布曲线差值的最大值。\n",
    "\n",
    "![](./data/KS_curve.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import layers,models,losses,metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#函数形式的自定义评估指标\n",
    "@tf.function\n",
    "def ks(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, (-1,))\n",
    "    y_pred = tf.reshape(y_pred, (-1,))\n",
    "    length = tf.shape(y_true)[0]\n",
    "    \n",
    "    t = tf.math.top_k(y_pred, k=length, sorted=False)\n",
    "    y_pred_sorted = tf.gather(y_pred, t.indices)\n",
    "    y_true_sorted = tf.gather(y_true, t.indices)\n",
    "    \n",
    "    cum_positive_ratio = tf.truediv(\n",
    "        tf.cumsum(y_true_sorted), tf.reduce_sum(y_true_sorted))\n",
    "    cum_negative_ratio = tf.truediv(\n",
    "        tf.cumsum(1 - y_true_sorted), tf.reduce_sum(1 - y_true_sorted))\n",
    "    ks_value = tf.reduce_max(tf.abs(cum_positive_ratio - cum_negative_ratio))\n",
    "    return ks_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.625\n"
     ]
    }
   ],
   "source": [
    "y_true = tf.constant([[1],[1],[1],[0],[1],[1],[1],[0],[0],[0],[1],[0],[1],[0]])\n",
    "y_pred = tf.constant([[0.6],[0.1],[0.4],[0.5],[0.7],[0.7],[0.7],\n",
    "                      [0.4],[0.4],[0.5],[0.8],[0.3],[0.5],[0.3]])\n",
    "tf.print(ks(y_true,y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#类形式的自定义评估指标\n",
    "class KS(metrics.Metric):\n",
    "    \n",
    "    def __init__(self, name = \"ks\", **kwargs):\n",
    "        super(KS,self).__init__(name=name,**kwargs)\n",
    "        self.true_positives = self.add_weight(\n",
    "            name = \"tp\",shape = (101,), initializer = \"zeros\")\n",
    "        self.false_positives = self.add_weight(\n",
    "            name = \"fp\",shape = (101,), initializer = \"zeros\")\n",
    "   \n",
    "    @tf.function\n",
    "    def update_state(self,y_true,y_pred):\n",
    "        y_true = tf.cast(tf.reshape(y_true,(-1,)),tf.bool)\n",
    "        y_pred = tf.cast(100*tf.reshape(y_pred,(-1,)),tf.int32)\n",
    "        \n",
    "        for i in tf.range(0,tf.shape(y_true)[0]):\n",
    "            if y_true[i]:\n",
    "                self.true_positives[y_pred[i]].assign(\n",
    "                    self.true_positives[y_pred[i]]+1.0)\n",
    "            else:\n",
    "                self.false_positives[y_pred[i]].assign(\n",
    "                    self.false_positives[y_pred[i]]+1.0)\n",
    "        return (self.true_positives,self.false_positives)\n",
    "    \n",
    "    @tf.function\n",
    "    def result(self):\n",
    "        cum_positive_ratio = tf.truediv(\n",
    "            tf.cumsum(self.true_positives),tf.reduce_sum(self.true_positives))\n",
    "        cum_negative_ratio = tf.truediv(\n",
    "            tf.cumsum(self.false_positives),tf.reduce_sum(self.false_positives))\n",
    "        ks_value = tf.reduce_max(tf.abs(cum_positive_ratio - cum_negative_ratio)) \n",
    "        return ks_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.625\n"
     ]
    }
   ],
   "source": [
    "y_true = tf.constant([[1],[1],[1],[0],[1],[1],[1],[0],[0],[0],[1],[0],[1],[0]])\n",
    "y_pred = tf.constant([[0.6],[0.1],[0.4],[0.5],[0.7],[0.7],\n",
    "                      [0.7],[0.4],[0.4],[0.5],[0.8],[0.3],[0.5],[0.3]])\n",
    "\n",
    "myks = KS()\n",
    "myks.update_state(y_true,y_pred)\n",
    "tf.print(myks.result())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
